Optimization in Knowledge-Intensive Crowdsourcing

We present SmartCrowd, a framework for optimizing collaborative knowledge-intensive crowdsourcing. SmartCrowd distinguishes itself by accounting for human factors in the process of assigning tasks to workers. Human factors designate workers' expertise in different skills, their expected minimum wage, and their availability. In SmartCrowd, we formulate task assignment as an optimization problem, and rely on pre-indexing workers and maintaining the indexes adaptively, in such a way that the task assignment process gets optimized both qualitatively, and computation time-wise. We present rigorous theoretical analyses of the optimization problem and propose optimal and approximation algorithms. We finally perform extensive performance and quality experiments using real and synthetic data to demonstrate that adaptive indexing in SmartCrowd is necessary to achieve efficient high quality task assignment.Comment: 12 page

Engineering Crowdsourced Stream Processing Systems

A crowdsourced stream processing system (CSP) is a system that incorporates crowdsourced tasks in the processing of a data stream. This can be seen as enabling crowdsourcing work to be applied on a sample of large-scale data at high speed, or equivalently, enabling stream processing to employ human intelligence. It also leads to a substantial expansion of the capabilities of data processing systems. Engineering a CSP system requires the combination of human and machine computation elements. From a general systems theory perspective, this means taking into account inherited as well as emerging properties from both these elements. In this paper, we position CSP systems within a broader taxonomy, outline a series of design principles and evaluation metrics, present an extensible framework for their design, and describe several design patterns. We showcase the capabilities of CSP systems by performing a case study that applies our proposed framework to the design and analysis of a real system (AIDR) that classifies social media messages during time-critical crisis events. Results show that compared to a pure stream processing system, AIDR can achieve a higher data classification accuracy, while compared to a pure crowdsourcing solution, the system makes better use of human workers by requiring much less manual work effort

Semantic Indexing and Retrieval based on Formal Concept Analysis

Semantic indexing and retrieval has become an important research area, as the available amount of information on the Web is growing more and more. In this paper, we introduce an original approach to semantic indexing and retrieval based on Formal Concept Analysis. The concept lattice is used as a semantic index and we propose an original algorithm for traversing the lattice and answering user queries. This framework has been used and evaluated on song datasets

Crowds, not Drones: Modeling Human Factors in Interactive Crowdsourcing

International audienceIn this vision paper, we propose SmartCrowd, an intelligent and adaptive crowdsourcing framework. Contrary to existing crowdsourcing systems, where the process of hiring workers (crowd), learning their skills, and evaluating the accuracy of tasks they perform are fragmented, siloed, and often ad-hoc, SmartCrowd foresees a paradigm shift in that process, considering unpredictability of human nature, namely human factors. SmartCrowd offers opportunities in making crowdsourcing intelligent through iterative interaction with the workers, and adaptively learning and improving the underlying processes. Both existing (majority of which do not require longer engagement from volatile and mostly non-recurrent workers) and next generation crowdsourcing applications (which require longer engagement from the crowd) stand to benefit from SmartCrowd. We outline the opportunities in SmartCrowd, and discuss the challenges and directions, that can potentially revolutionize the existing crowdsourcing landscape

Crowds, not Drones: Modeling Human Factors in Interactive Crowdsourcing

International audienceIn this vision paper, we propose SmartCrowd, an intelligent and adaptive crowdsourcing framework. Contrary to existing crowdsourcing systems, where the process of hiring workers (crowd), learning their skills, and evaluating the accuracy of tasks they perform are fragmented, siloed, and often ad-hoc, SmartCrowd foresees a paradigm shift in that process, considering unpredictability of human nature, namely human factors. SmartCrowd offers opportunities in making crowdsourcing intelligent through iterative interaction with the workers, and adaptively learning and improving the underlying processes. Both existing (majority of which do not require longer engagement from volatile and mostly non-recurrent workers) and next generation crowdsourcing applications (which require longer engagement from the crowd) stand to benefit from SmartCrowd. We outline the opportunities in SmartCrowd, and discuss the challenges and directions, that can potentially revolutionize the existing crowdsourcing landscape

Using pattern structures to support information retrieval with Formal Concept Analysis

International audienceIn this paper we introduce a novel approach to information retrieval (IR) based on Formal Concept Analysis (FCA). The use of concept lattices to support the task of document retrieval in IR has proven effective since they allow querying in the space of terms modelled by concept intents and navigation in the space of documents modelled by concept extents. However, current approaches use binary representations to illustrate the relations between documents and terms (''document D contains term T'') and disregard useful information present in document corpora (''document D contains X references to term T''). We propose using pattern structures, an extension of FCA on multi-valued and numerical data, to address the above. Given a set of weighted document-term relations, a concept lattice based on pattern structures is built and explored to find documents satisfying a given user query. We present the meaning and capabilities of this approach, as well as results of its application over a classic IR document corpus

Unleashing the Potential of Crowd Work: The Need for a Post-Taylorism Crowdsourcing Model

Paid crowdsourcing connects task requesters to a globalized, skilled workforce that is available 24/7. In doing so, this new labor model promises not only to complete work faster and more efficiently than any previous approach but also to harness the best of our collective capacities. Nevertheless, for almost a decade now, crowdsourcing has been limited to addressing rather straightforward and simple tasks. Large-scale innovation, creativity, and wicked problem solving are still largely out of the crowd’s reach. In this opinion paper, we argue that existing crowdsourcing practices bear significant resemblance to the management paradigm of Taylorism. Although criticized and often abandoned by modern organizations, Taylorism principles are prevalent in many crowdsourcing platforms, which employ practices such as the forceful decomposition of all tasks regardless of their knowledge nature and the disallowing of worker interactions, which diminish worker motivation and performance. We argue that a shift toward post-Taylorism is necessary to enable the crowd address at scale the complex problems that form the backbone of today’s knowledge economy. Drawing from recent literature, we highlight four design rules that can help make this shift, namely, endorsing social crowd networks, encouraging teamwork, scaffolding ownership of one’s work within the crowd, and leveraging algorithm-guided worker self-coordination.Peer Reviewedhttp://deepblue.lib.umich.edu/bitstream/2027.42/171075/1/Lykourentzou et al. 2021.pdfDescription of Lykourentzou et al. 2021.pdf : Final ArticleSEL